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ABSTRACT 

One of the major program goals of the University 
Council for Educational Administration (UCEA) is the design, 
development, and production of a series of simulation materials for 
the preservice and inservice training of urban educational 
administrators. This paper presents evaluative information regarding 
the Janus Junior High Principalship Simulation, the first simulation 
to be completed. The school is an actual inner-city junior high 
school located in Monroe City, North Columbia. Components of the 
urban simulation (UR3SIM) project are listed. Data collected both 
from participants in brief introductory dissemination institutes and 
from those who participated in full-length training workshops are 
presented in evaluations of selected components of the Janus URBSIM 
materials. The study indicated that the current UCEA dissemination 
strategy of introducing simulation materials by means of 
university-sponsored institutes and of collecting materials 
evaluation data at those institutes is an effective dissemination 
strategy. (DB) 
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PARTICIPANT EVALUATION OF UCEA URBAN SIMULATION MATERIALS AND WORKSHOPS 



One or the major program goals of the University Council for Educational Administration 
(UCEA) for the five year period 1969-74 is the design, development, and production of a 
series of simulation materials for the preservice and inservice training of urban educational 
administrators. Supported in part by grants from the U. S. Office of Education, the urban 
simulation (URBSIM) project represents a unique example in American education of 
the "critical mass” principle in development. 1 More than 80 professors from about 
40 universities and a number of others from related organizations have thus far been 
involved in developmental work associated with the URBSIM project. Another 100 educators 
from nearly 50 universities have been involved in demonstrating and disseminating 
URBSIM products, as they have become available, in regional university-sponsored 
institutes. Three sets of simulations — the Janus Junior High School Principal ship, the 
Wilson Senior High School Principals hip, and the Abraham Lincoln Elementary School 
Principalship — have thus far been completed. A number of additional role and process 
simulations are presently in various stages of development. 

The purpose of this paper is to present evaluative information regarding the Janus 
Junior High Principalship Simulation, the first simulation to be completed. Data has 
been gathered both from participants in brief introductory dissemination institutes and 
from those who have participated in full-length training workshops. 
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This information has been sought for the purpose of assisting developers 
UCEA Central Staff and on simulation project development teams to create more 
effective instructional materials in ongoing and future simulation developme 

tlme in the BKBSiM prqjeet, one of the basic developmental principles in the over,l program 

design has been to encourage each development team to build upon and .tend the wor 

, a uoroin is seen as a portion of the data 
of its predecessors. The information reported 

. . h „ ail of the information reported here was obtained 
necessary to achieve that objective. All of the 

from written questionnaires administered in institutes and workshops under 
controlled conditions. The data are entirely subjective, and the usual caveats « T- 

NonetheU.®. » ~ - — - - * — — — ** ** “ „ 

„ .. — • - — « — « " •— * - “‘“ 4 

Janus Junior Hifh Sc hool Princlpalshlp_SimnMiHl 

janus Junior High School is based upon and represents an actual inner-city Jun or 
Mgh school located in Monroe City. North Columbia. UBBS1M components of tie 

Janus Junior High School Principals!,* Simulation include the following: 

i T ibrarv 15 background booklets describing 

Us school system. 

' 2 . Background Film: -'Monroe City" (30 minutes, color). 

3 * 

High School: The School. ” 

4, Janus Student Handbook. 
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5. Janus Data Bank: about 150 pages of material replicating that found in 

file cabinets in the Janus Principal’s office; includes 3 audio tapes. 

6. Five filmed critical incidents (20 minutes, black and white). 

7. Three filmed critical incidents (10 minutes, color). 

8. Janus In-Basket I: 22 decision items. 

9. Janus In-Basket II: 22 additional decision items. 

10. Audio taped interruptions. 

11. Janus expendable materials. 

12. Janus principalship instructor’s manual. 

13. Suggestions for use of the UCEA Monroe City simulations. 

14. Packet of specimen evaluation and response forms. 

The Janus simulation was introduced at 15 regional university-sponsored UCEA 

institutes in the United States and Canada in early 1971. Among the several purposes 

of the Janus institutes was the aggregation of critical evaluations of the materials by 

those who were in attendance at these two day institute sessions. An evaluation 

questionnaire was designed by Alan K. Gaynor, then Associate Director of UCEA and 

now Associate Professor of Educational Administration at Boston University, and was 

o 

administered by institute directors to 236 participants at 10 Janus institutes. 

Participant Evaluation of the Janus 
Simulation; Institutes vs. Workshops 

The Janus Institutes were relatively short in duration and were designed primarily 
to introduce the materials, rather than to use them for training. Therefore, it was 
decided that a second set of participant evaluations of the Janus materials reflecting actual 

o 
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workshop training use should be sought. From the development standpoint, a major 
purpose of the second study was to gain an indication of how critical evaluations of the 
materials obtained during introductory institutes compared with critical evaluations 
obtained during actual use by student participants. The second study was conducted 
during the summer of 1971. Questionnaires similar to the Gaynor instrument were 
administered to 96 workshop participants in 6 separate workshops. The typical summer 
workshop included in this study was of about 35 hours in length, with the training focused 
solely on the Janus Principalship role. 

Institute and workshop participant evaluations of selected components of the Janus 
URBSIM materials are presented below. 

Comparison of Institute and Workshop 
Evaluation of Janus Simulation Components 

(Institutes included in this summary (n=10): Alberta, Berkeley, Buffalo, Colorado, 

Connecticut, Houston, Illinois, Minnesota, William and Mary, one unidentified. 

N for Institute participants = 236. Workshops included in this summary (n=6): Buffalo, 

Cornell, Houston, Tennessee (2), and Plattsburgh. N for workshop participants = 96. 

Because all Janus components were not used in all institutes and workshops, n for 

participants varies.) 

INSTITUTES WORKSHOPS 

1, The background film, M Monroe City" 
a. Technical Quality 

= 86 (38%) 

= 132 (58%) 

= 8 ( 4%) 



(1) outstanding 

(2) good 

(3) poor 



28 (36%) 
48 (62%) 
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INSTITUTES 


WORKSHOPS 


b. Interest 


(1) outstanding 


- 40 (18%) 


26 (34%) 


(2) good 


= 150 (67%) 


47 (61%) 


(3) poor 


= 35 (15%) 


4 ( 5%) 


c. Giving a "feel" for the City 


(1) outstanding 


= 29 (13%) 


20 (26%) 


(2) good 


= 114 (51%) 


51 (67%) 


(3) poor 


= 80 (36%) 


5 ( 7%) 


2. "The Attendance Area" Slide Presentation 


a. Technical Quality 






(1) outstanding 


= 99 (45%) 


46 (48%) 


(2) good 


= 116 (52%) 


50 (52%) 


(3) poor 


= 7(3%) 


0 ( 0%) 


b. Interest 


(1) outstanding 


= 89 (40%) 


46 (48%) 


(2) good 


* 128 (57%) 


50 (52%) 


(3) poor 


= 8 ( 4%) 


0 ( 0%) 


c. Value As a Source of Information for the Principal of Janus 




(1) outstanding 


= 106 (48%) 


46 (48%) 


(2) good 


= 107 (48%) 


50 (52%) 


(3) poor 


* 8 ( 4%) 


0 ( 0%) 


3. "The School" Slide Presentation 


< 




a. Technical Quality 


(1) outstanding 


= 75 (33%) 


29 (33%) 


(2) good 


= 138 (62%) 


59 (67%) 


(3) poor 


= 11 ( 5%) 


0 ( 0%) 
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INSTITUTES 



b. Interest 

(1) outstanding 

(2) good 

(3) poor 



« 73 (33%) 

= 142 (64%) 
= 7 ( 3%) 



c. Value As a Source of Information for the Principal 



(1) outstanding 

(2) good 

(3) poor 



= 75 (34%) 

= 133 (60%) 
- 12 ( 6 %) 



4 . The Kinescopes lb Black and White filmed critical Incidents 



a. Overall Technical Quality - All Incidents 



(1) outstanding 

(2) good 

(3) poor 


= 6(3%) 

= 42 (19%) 

= 176 (78%) 


The Drug Scene 




(1) Believability 

(a) outstanding 

(b) good 

(c) poor 

(2) Utility for Instruction 

(a) outstanding 

(b) good 

(c) poor 


= 24 (10%) 

= 143 (57%) 
= 83 (33%) 

= 27 (12%) 

- 137 (61%) 
= 59 (27%) 


The Student Cafeteria Boycott 




(1) Believability 

(a) outstanding 

(b) good 

(c) poor 

(2) Utility for Instruction 

(a) outstanding 

(b) good 

(c) poor 


= 28 (12%) 
«= 136 (60%) 
= 62 (28%) 

= 33 (15%) 

= 133 (59%) 
= 58 (26%) 




WORKSHOPS 



37 (40%) 
55 (60%) 
0 ( 0%) 



46 (50%) 
45 (49%) 
1 ( 1 %) 



27 (32%) 
43 (56%) 
10 ( 12 %) 



29 (50%) 
21 (36%) 
8 (14%) 

33 (55%) 
21 (35%) 
6 ( 10 %) 



24 (41%) 
26 (44%) 
9 (15%) 

24 (42%) 
31 (53%) 
3 ( 5%) 
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Black Student- White Teacher Confrontation 


INSTITUTES 


WORKSHOPS 


(1) Believability 


(a) outstanding 


= 82 (36%) 


44 (67%) 


(b) good 


= 124 (55%) 


22 (33%) 


(c) poor 


= 19 ( 9%) 


0 ( 0%) 


(2) Utility for Instruction 


(a) outstanding 


= 68 (30%) 


45 (68%) 


(b) good 


= 131 (59%) 


21 (32%) 


(c) poor 


* 24 (11%) 


0 ( 0%) 


Black Students-Principal Confrontation 
(1) Believability 


(a) outstanding 


= 79 (34%) 


39 (42%) 


(b) good 


» 132 (58%) 


53 (58%) 


(c) poor 


= 18 ( 8%) 


0 ( 0%) 


(2) Utlity for Instruction 


(a) outstanding 


= 92 (41%) 


55 (58%) 


(b) good 


= 123 (54%) 


39 (41%) 


(c) poor 


= 12 ( 5%) 


1 ( 1%) 


Teacher Strike 






(1) Believability 


(a) outstanding 


= 52 (23%) 


10 (28%) 


(b) good 


= 131 (57%) 


21 (58%) 


(c) poor 


= 47 (20%) 


5 (14%) 


(2) Utility for Instruction 


(a) outstanding 


= 37 (16%) 


17 (47%) 


(b) good 


= 129 (57%) 


14 (39%) 


(c) poor 


= 62 (27%) 


5 (14%) 



5. The Three Color Filmed Critical Incidents 
a. Overall Technical Quality - All Inci dents 



* 118 (51%) 

* 110 (48%) 

= 2 ( 1 %) 



39 (48%) 

40 (49%) 
2 ( 3%) 




(1) outstanding 

(2) good 

(3) poor 
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INSTITUTES 


WORKSHOPS 


"The Outsider” (non-student visitor and teacher in the hall) 




(1) Believability 






(a) outstanding 


= 109 (48%) 


42 (58%) 


(b) good 


= 101 (44%) 


28 (38%) 


(c) poor 


= 17 ( 8%) 


3 ( 4%) 


(2) Utility for Instruction 






(a) outstanding 


= 95 (41%) 


44 (61%) 


(b) good 


= 123 (54%) 


26 (36%) 


(c) poor 


- 11 ( 5%) 


2 ( 3%) 


"Outside Advice" (citizens’ 


committee requests police in the school) 


(1) Believability 




35 (41%) 


(a) outstanding 


= 87 (38%) 


(b) good 


= 108 (47%) 


46 (54%) 


(c) poor 


= 33 (15%) 


4 ( 5%) 


(2) Utility for Instruction 






(a) outstanding 


= 111 (49%) 


51 (62%) 


(b) good 


= 104 (46%) 


31 (37%) 


(c) poor 


= 12 ( 5%) 


1 ( 1%) 



d. "A Sincere Proposal** (for black and white faculty to meet socially) 



(1) Believability 



(a) outstanding 


= 55 (24%) 


15 (22%) 


(b) good 


= 136 (59%) 


42 (63%) 


(c) poor 


= 3p (17%) 


10 (15%) 


(2) Utility for Instruction 






(a) outstanding 


= 46 (19%) 


16 (31%) 


(b) good 


* 133 (56%) 


31 (61%) 


(c) poor ^ 


= 59 (25%) 


4 ( 8%) 



The foregoing data indicate, in general, that institute audiences tend to be 
more critical of simulation materials than workshop audiences. This is a significant 
finding in terms of development and dissemination strategy, for it suggests that simulation 
materials which pass the test of introduction through university-sponsored institutes will 
be favorably received when they are later used in actual instructional situations. 



o 
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The data reflect two major decisions about Janus components which were made as a 
result of Gaynor's institute feedback. The first decision had to do with the ’’Monroe City” 
color film. As institute directors proceeded through their schedule of 15 Janus institutes, 
it became clear that a carefully conceived instructional strategy would be necessary to 
make effective use of the Monroe City film in teaching situations. Such a strategy was 
designed and implemented by the workshop directors during the summer workshops. It 
appears likely that the newly developed instructional strategy relating to the ’’Monroe City" 
film would account in large part for the improved ratings of that component in workshops 
as compared to institutes. 

In the second instance, the low ratings accorded to the five black and white filmed 
incidents on the technical quality criterion resulted in a decision to completely re-shoot those 
incidents. The substantially higher ratings accorded to the five re-photographed black 
and white filmed incidents in the workshop setting would indicate that this effort justified the 
additional development cost involved in re-staging and producing an improved version of 
the film. 

Workshop Participant Evaluation of Janus 
Components Not Evaluated at Institutes 

Because of the short duration of the Janus Institutes, a number of the Janus written 
components could not be evaluated by institute participants. In the more extensive time 
period available for workshop instruction, however, it was possible for workshop participants 
to evaluate these particular written components of the simulation. Simulation components 
in this category include the Monroe City Background Libre rt , Janus In-Basket I, 
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Janus In-Basket II, the Janus Data Bank, and a series of structured feedback instruments. 
The structured feedback instruments were designed to fill, at least in part, a 

* 

recognized shortcoming which has existed in most education simulation ejqjeriences. 

The need to be met here is the provision to workshop participants of conceptually 
organized feedback information regarding their own behavior and performance. A number 
of pilot participant feedback instruments were developed during the spring of 1971 for use 
in the Wilson Senior High School principalship instructional materials institutes. As these 
pilot structured feedback instruments were designed to be used with all three URBSIM 
principalship simulations, they were available for use in the Janus workshops during the 
summer of 1971. The structured feedback instruments so used include (1) The Value 
Resolution Scale, (2) The Action Analysis Profile, (3) The Means of Communication Profile, 
and (4) The Value Assumptions Profile.** 

A summary of workshop participant evaluations of these written materials is presented 
below. 



Workshop Participant Evaluation of Various Janus Simulation Components 

* 

1. Evaluation of the Monroe City Background Library booklets: 



(1) excellent 




55 (60%) 


(2) good 


55 


36 (40%) 


(3) fair 


*5 


0 ( 0%) 


(4) poor 


55 


0 ( 0%) 


of the In-Basket I materials: 


(1) excellent 


-5 


65 (68%) 


(2) good 


s 


31 (32%) 


(3) fair 


S5 


0 ( 0%) 


(4) poor 


S 


0 ( 0%) 
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3. Evaluation of the In-Basket II materials: 



(1) excellent 


ss 


61 (64%) 


(2) good 


55 


33 (34%) 


(3) fair 


55 


1 ( 1%) 


(4) poor 


2? 


1 ( 1%) 


Evt Illation of the Janus Data Bank: 


a. (1 (1) too extensive 


SS 


2 ( 2%) 


(2) about right 


5= 


62 (65%) 


(3) too limited 


53 


32 (33%) 


b. (1) highly useful 


S3 


36 (38%) 


(2) of some use 


= 


87 (69%) 


(3) of little use 


S3 


3 ( 3%) 
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5. The Value Resolution Scale (administered twice as a pre- and post-simulation 
instrument): 



a. First (Pre-Simulation) Administration 





(1) highly useful 


32 


21 (28%) 




(2) of some use 


3= 


87 (68%) 




(3) of little use 


25 


6 ( 7%) 




b. Second (Post- Simulation) Administration 


- 




(1) highly useful 


35 


43 (63%) 




(2) of some use 


23 


28 (38%) 




(3) of little use 

% 


sr 


10 (12%) 


6. 


The Action Analysis Profile: 








(1) highly useful 


35 


38 (48%) 




(2) of some use 


SS 


41 (61%) 




(3) of little use 


52 


1 ( 1%) 


7. 


The Means of Communication Profile: 








(1) highly useful 


= 


34 (42%) 




(2) of some use 


22 


41 (61%) 




(3) of little use 


SS 


6 ( 7%) 


8. 


The Value Assumptions Profile: 








(1) highly useful 


r: 


22 (28%) 




(2) of some use 




48 (68%) 




(3) of little use 


= 


11 (14%) 
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The foregoing data indicate that the Monroe City Background Library and Janus 
In-Baskets I and n were well received in the six workshops, with about two-thirds of 
the participants (60% to 68%) rating them ’’excellent” and about one-third (32% to 40%) 
rating them "good. ” About two-thirds (65%) of the respondents rated the Janus Data Bank 
as "about right" in comprehensiveness, with most of the remaining respondents (33%) 
finding it "too limited, " A minority of the respondents (38%) rated the Data Bank as 
"highly useful," however, with most (59%) rating it only as "of some use. " 

Some variation is shown in ratings accorded the structured feedback instruments. 

Over half (53%) found the post- simulation administration of the Value Resolution Scale 
"highly useful," and nearly half (48%) accorded the Action Analysis Profile a "highly useful" 
rating. The lowest rating was placed on the Value Assumptions Profile, rated by 58% as "of some 
use" and by 14% as "of little use. " 

Evaluation of the Workshop Experience 

The Monroe City URBSIM materials are the most extensive and complex simulation 
exercises which have thus far been developed for the inservice and preservice preparation 
of educational administrators. Earlier simulations, such as the UCEA Jefferson and 
Madison materials, have usually been viewed as useful and satisfying experiences by 
participants, as professors of educational administration who have used those materials 
will attest. (It should be noted that we are here speaking of subjective evaluation of the 
simulation experience itself, as distinct from measured behavioral change or the impact of 
simulation : raining on post- simulation performance. These latter issues are beyond the 
scope of this report. ) Some of those involved in development of the Monroe City principalship 
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simulations wondered, on occasion, whether the materials they were developing were too 
"demanding,” too "tough, " or too frustrating to yield the affective rewards to participants 
which have come to be associated with simulation experiences. For this reason, workshop 
participants were asked to respond to a series of items related to the Janus prineipalship 
workshop experience. The item stems and response frequencies are presented below. 



Evaluation of the Janus Workshop Experience 
1. In this simulation, I learned most from: 4 



(1) 


the materials 


= 25 (18%) 


(2) 


other participants 


= 74 (53%) 


( 3 ) 


observing my own behavior 


= 32 (23%) 


( 4 ) 


other 


= 8 ( 6%) 



2 . 



This workshop would be most useful for: 



4 



(1) experienced administrators 

(2) beginning administrators 

(3) those aspiring to be administrators 

(4) teachers 

(5) other 



34 (20%) 
71 (43%) 
44 (26%) 
13 ( 8%) 
5 ( 3%) 



3. In general, I was able to identify with Leslie Bunker (principal of Janus): 



(1) 


very well 


= 16 (16%) 


(2) 


rather well 


= 51 (54%) 


( 3 ) 


somewhat 


= 27 (29%) 


( 4 ) 


not at all 


= 1(1%) 



4. The "urban reality" of the simulation was: 



(1) 


highly convincing 


= 50 (53%) 


(2) 


moderately convincing 


= 45 (47%) 


( 3 ) 


unconvincing 


= 0(0%) 
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5. Compared to other courses, seminars, and workshops I have had in educational 

administration, the Janus simulation was; 

(1) the best * 49 (53%) 

(2) better than most = 36 (39%) 

(3) about average - 8 ( 8%) 

(4) worse than most = 0 ( 0%) 

(5) the worst = 0 ( 0%) 

6. As a result of this workshop, my desire to become or remain an administrator 

has been: 



(1) increased = 46 (48%) 

(2) not changed = 48 (50%) 

(3) reduced = 2 ( 2%) 

As these data indicate, more than 9 out of 10 (92%) participants evaluated the Janus 

simulation experience as better than most other preparatory program work, with over 

/ 

half (53%) ranking it as "the best. " Few participants (2%) were apparently deterred from the 
principalship by the workshop experience. Over half (53%) of the participants felt that they 
learned most from other participants, and 7 out of 10 (70%) felt that they were able to 
identify with the Janus principalship role (Leslie Bunker) "rather well" or "very well." 
Conclusion 

In summary, this study indicates that the current UCEA dissemination strategy of 
introducing simulation materials by means of university-sponsored institutes and of 
collecting materials evaluation data at those institutes is an effective dissemination strategy. 
Since institute audiences appear to be more critical of simulation components than workshop 
participants, it may be suggested that simulation components which successfully meet 
institute evaluation standards will be seen as effective when used for actual instructional 
purposes in workshops. 
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The generally favorable evaluation of the structured participant feedback instruments 
would appear to support and encourage further development work on these materials. 

And, finally, the Janus simulation experience was seen by participants as valuable and 
satisfying by nearly all participants, with 92% rating it as ’’better than most” or ’’the best" 
educational administration program component which they had experienced. 
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